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DIGITAL WATERMARKING 

FIELD OF INVENTION 

The present invention relates to digital watermarking of 
data including image, video and multimedia data. 
Specifically, the invention relates to the insertion and extrac- 
tion of embedded signals for purposes of watermarking, in 
which the insertion and extraction procedures are repeatedly 
applied to subregions of the data. When these sub regions 
correspond to the 8x8 pixel blocks used for MPEG and 
JPEG compression and decompression, the watermarking 
procedure can be tightly coupled with these compression 
algorithms to achieve very significant savings in computa- 
tion. 

BACKGROUND OF THE INVENTION 

The proliferation of digitized media such as image, video 
and multimedia is creating a need for a security system 
which facilitates the identification of the source of the 
material. 

Content providers, i.e. owners of works in digital data 
form, have a need to embed signals into video/image/ 
multimedia data which can subsequently be detected by 
software and/or hardware devices for purposes of authenti- 
cating copyright ownership, control and management. 

For example, a coded signal might be inserted in data to 
indicate that the data should not be copied. The embedded 
signal should preserve the image fidelity, be robust to 
common signal transformations and resistant to tampering. 
In addition, consideration must be given to the data rate that 
can be provided by the system, though current requirements 
are relatively low — a few bits per frame. 

In U.S. patent application Ser. No. 08/534,894, filed Sep. 
28, 1995,. entitled "Secure Spread Spectrum Watermarking 
for Multimedia Data" now abandoned and assigned to the 
same assignee as the present invention, which is incorpo- 
rated herein by reference, there was proposed a spread 
spectrum watermarking method which embedded a water- 
mark signal into perceptually significant regions of an image 
for the purposes of identifying the content owner and/or 
possessor. A strength of this approach is that the watermark 
is very difficult to remove. In fact, this method only allows 
the watermark to be read if the original image or data is 
available for comparison. This is because the original spec- 
trum of the watermark is shaped to that of the image through 
a non -linear multiplicative procedure and this spectral shap- 
ing must be removed prior to detection by matched filtering 
and the watermark is inserted into the N largest spectral 
coefficients, the ranking of which is not preserved after 
watermarking. Thus, this method does not allow software 
and hardware devices to directly read embedded signals. 

In an article by Cox et al., entitled "Secured Spectrum 
Watermarking for Multimedia" available at http:// 
www.neci.nj.com/tr/index.html (Technical Report No. 
95-10) spread spectrum watermarking is described which 
embeds a pseudo-random noise sequence into the digital 
data for watermarking purposes. 

The above prior art watermark extraction methodology 
requires the original image spectrum be subtracted from the 
watermark image spectrum. This restricts the use of the 
method when there is no original image or original image 
spectrum available. One application where this presents a 
significant difficulty is for third party device providers 
desiring to read embedded information for operation or 
denying operation of such a device. 
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In U.S. Pat. No. 5,319,735 by R. D. Preuss et al entitled 
"Embedded Signalling" digital information is encoded to 
produce a sequence of code symbols. The sequence of code 
symbols is embedded in an audio signal by generating a 

5 corresponding sequence of spread spectrum code signals 
representing the sequence of code symbols. The frequency 
components of the code signal being essentially confined to 
a preselected signaling band lying within the bandwidth of 
the audio signal and successive segments of the code signal 

10 corresponds to successive code symbols in the sequence. 
The audio signal is continuously frequency analyzed over a 
frequency band encompassing the signalling band and the 
code signal is dynamically filtered as a function of the 
analysis to provide a modified code signal with frequency 

15 component levels which are, at each time instant, essentially 
a preselected proportion of the levels of the audio signal 
frequency components in corresponding frequency ranges. 
The modified code signal and the audio signal are combined 
to provide a composite audio signal in which the digital 

20 information is embedded. This component audio signal is 
then recorded on a recording medium or is otherwise sub- 
jected to a transmission channel. Two key elements of this 
process are the spectral shaping and spectral equalization 
that occur at the insertion and extraction stages, respectively, 

25 thereby allowing the embedded signal to be extracted with- 
out access to the unwatermarked original data. 

In U.S. patent application Ser. No. 08/708,331, filed Sep. 
4, 1996,entitled "A Spread Spectrum Watermark for Embed- 
ded Signaling" by Cox; now U.S. Pat. No. 5,848,155 and 

30 incorporated herein by reference, there is described a 
method for extracting a watermark of embedded data from 
watermarked images or video without using an original or 
unwatermarked version of the data. This work can be viewed 
as an extension of the original work of Preuss et al from the 

35 audio domain to images and video. 

This method of watermarking an image or image data for 
embedding signaling requires that the DCT (discrete cosine 
transform) and its inverse of the entire image be computed. 
There are fast algorithms for computing the DCT in N log 

40 N time, where N is the number of pixels in the image. 
However, for N«5 12x512, the computational requirement is 
still high, particularly if the encoding and extracting pro- 
cesses must occur at video rates, i.e. 30 frames per second. 
This method requires approximately 30 times the computa- 

45 tion needed for MPEG-II decompression. 

One possible way to achieve real-time video watermark- 
ing is to only watermark every N th frame. However, content 
owners wish to protect each and every video frame. 
Moreover, if it is known which frames contain embedded 

50 signals, it is simple to remove those frames with no notice- 
able degradation in the video signal. 

In U.S. patent application Ser. No. 08/715,953, filed Sep. 
19, 1996,entitled "Watermarking of Image Data Using 

55 MPEG/SPEG Coefficients" by Cox, and incorporated herein 
by reference, there is described an alternative method, which 
is to insert the watermark into nxn blocks of the image 
(subimages) where n«N. Then the computation cost is 

60 —n logn = N logn. 

For N=512x512«2 18 and n=8x8»2 6 , the asymptotic sav- 
ing is only a factor of 3. However, empirically the cost of 
65 computing the DCT over the entire image may be signifi- 
cantly higher when cache, loop unfolding and other effi- 
ciency issues are considered. Thus, the practical difference 
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may approach a 30 fold savings. More importantly, if the FIG. 6 is a graphic representation of rotation of PN 

block size is chosen to be 8x8, i.e. the same size as that used sequences; 

for MPEG image compression, then it is possible to tightly FIG. 7 is a graphical representation of an 8x8 block 

couple the watermark insertion and extraction procedures to shown the spatial relation of averaged terms; 
those of the MPEG compression and decompression algo- 5 FIG. 8 is a schematic block diagram of a method for 

rithms. Considerable computational saving can then be inserting watermarks in accordance with the present inven- 

achieved since the most expenses computations relate to the tion; and 

calculation of the DCT and its inverse and these steps are FIG. 9 is a schematic block diagram of a method for 

already computed as part of the compression and decom- extracting watermarks in accordance with the present inven- 

pression algorithm. The incremental cost of watermarking is 10 lion. 

then very small, typically less than 5% of the computational DETAILED DESCRIPTION 

requirements associated with MPEG. n *. . , ^ ... 

„, 4 . - iL Referring now to the figures, and FIGS. 1 through 4 in 

_ Ine present invenUon improves the reliability ot the particular T here 

is shown schematic block diagrams of a 
invention described in the 08/715,953 application, now , method fof inserti and de[ectj wa f ermarks in 

pending by storing watermark information into subimages, " fof . 

and extracting watermark information from subimages, m a » 4 . r « • j r L 

manner different from that described earlier. . In th f , foUowln & de ^ tl0 °' refere " ce ™ y ^J™*? 0 

image data or images. While the invention has applicability 

SUMMARY OF THE INVENTION to ^ ma § e data m & images, it will be understood that the 

2 q teachings herein and the invention itself are equally appli- 

The present invention improves the reliability of the prior cable to video, image and multimedia data and the term 

systems by systematically varying the order in which water- "image" and "image data" will be understood to include 

mark signal components are inserted into each subimage, by these terms where applicable. As used herein, "watermark" 

inserting only part of the watermark signal into each will be understood to include embedded data, symbols, 

subimage, and, during watermark detection, by combining 25 images, instructions or any other identifying information, 
the watermark signals found in groups of subimages to In the following description, reference is made to proce- 

reconstruct the original watermark signal before testing for dures described in U.S. patent application Ser. No. 08/534, 

correlation with any predefined watermarks. 894 for inserting and extracting or detecting a watermark in 

For detection, a reverse transformation is applied to each images as INSERT-ORIGINAL and EXTRACT- 

subimage to reconstruct the watermark information that was 30 ORIGINAL, respectively. Reference is made to procedures 

stored in that subimage. The resulting signals are then described in U.S. patent application Ser. No. 08/708,331 

averaged together to reconstruct the whole watermark, and filed Sep. 4, 1996,now U.S. Pat. No. 5,848,155 for inserting 

to reduce noise. Finally, this reconstructed watermark is and extracting or detecting watermarks in images as 

compared against a predefined set of watermark signals to INSERT- WHOLE and EXTRACT- WHOLE, respectively, 

determine which one was inserted into the image. 35 And reference is made to procedures described in U.S. 

Aprincipal object of the present invention is therefore, the P atent application Ser. No. 08/715,953 for inserting and 

provision of inserting a subset of a watermark into a subset extracting or detecting watermarks in images as INSERT- 

of subregions of data to be watermarked. MPEG-A and EXTRACT-MPEG-A, respectively. 

Another object of the invention is the provision of a ™L 1 show * a ^^ematic block diagram of INSERT- 

digital watermarking system in which a watermark is 40 ™ LE procedure inserting watermarks into images, 

extracted by averaging the watermarked signal from subre- The watermark signal, in the form of a finite sequence of 

gions of watermarked data, and then correlating the resulting s y mhoh chosen from an «lph«bet, » provided as an input to 

signal to determine the watermark. an error correctlon encoder 10 which transforms this 

, - . . . r.. sequence into another sequence that contains redundant 

A further object of the invention is the provision of a 45 information . ^ oul t of encodef 10 is ided t0 a 

digital watermarking system in which the watermark is PN _ u which each 5o] of lne encoded 

composed of two portions, a verification portion and a watermark into a ^.^^ pS eudo-random noise (PN) 

synchronization portion, in order to improve watermark code ^ 0lUput Qf ^ pN _ mapper u is provided \ 0 [ 

extraction reliability spectral transformer 12, which converts the pseudo-random 

Further and still other objects of the invention will 50 sequence into the frequency domain. The conversion 

become more clearly apparent when the following descrip- preferably is by discrete cosine transform (DCT), however, 

tion is read in conjunction with the accompanying drawing. f ast fourier transform, wavelet type decomposition and the 

„„ ¥r ^ ^„„ nWTVr „^ r _ _ like may also be used for frequency conversion. 

BRIEF DESCRIPTION OF THE DRAWINGS Concurrently, the data to be watermarked is provided to 

FIG. 1 is a schematic block diagram of watermark inser- 55 another spectral transformer 13. The outputs of the two 

tion procedure* spectral transformers 12 and 13 are then provided as inputs 

FIG. 2 is a schematic block diagram of a watermark |° a s P«* ral sha P er 14 T which . mod * es ; he spectral P^pc:- 
insertion procedure in accordance with the teachings of the ' 1CS of P^™*«* noise codes from spectral trans- 
present invention* former 12 to mask the watermark when added to the image 
- . \ e 60 data. The spectrally transformed data to be watermarked, 

FIG, 3 is a schematic block diagram of a watermark from spectral transformer 13, is also provided as an input to 

extraction procedure; a delay 15 ^ outpm of tfae spectral shaper 14 ^ then added 

FIG. 4 is a schematic block diagram of a watermark to the output of delay 15 at a summer 16. The summer output 

extraction procedure in accordance with the teachings of the is subject to an inverse transform 17. The result of the 

present invention; 65 inverse transform is watermarked data. 

FIG. 5 is a graphic representation of a zigzag pattern INSERT-MPEG- A differs from INSERT- WHOLE by seg- 

useful for vectorizing subimages; menting the data to be watermarked into multiple blocks, 
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such as 8x8 pixel subimages or subregions. Each block of into blocks by data segmenter 40, which corresponds to the 

data then has the watermark inserted according to the above data segmenter 24 used during the insertion procedure in 

described method. That is, for each 8x8 subimage or FIG. 2. Each of the data blocks is provided to a respective 

subregion, a pseudo-random number (PN) sequence is spectrum normalizer 41a, 41b, etc. to produce a signal 

inserted into the DCT coefficients after suitable spectral s resembling the subwatermark that was inserted into the 

shaping. The procedure is repeated for all such subimages or respective data block. These inserted subwatermark signals 

subregions. The size of the subimage or subregion is pref- are , nen ^4 ^ mputs int0 a wa termark combiner 42. In the 

erably 8x8, but it can be of other sizes, such as 2x2, 3x3, 4x4 combiner 42 , parts of the watermark that appear redundantly 

or loxlo. ; n severa j subwatermarks are averaged together to reduce 

FIG. 2 shows a schematic block diagram of a watermark 10 output G f tne wa termark combiner 42 is provided 

insertion procedure in accordance with teachings of the as , he , 0 a symbo) 43 which divides the 

present invention. The watermark signal is processed into a watermark into parts, each of which corresponds to one 

noise spectrum signal by the error correction encoder 20, the symbol frora lhe encoded wa t er mark signal (the output of 

PN mapper 21, and the spectral transformer 22, in the same error correct i on encoder 20 in FIG. 2). 

m ^ ner ^o^ r i^ c^Tx™^ 1, u ° WeVer ' 15 These symbols from separator 43 are provided as inputs 

unlike INSERT-WHOLE or INSERT-MPEG-A, the water- , o ^ ^ ^ ^ ^ q[ 

mark is then used as an input to a watermark segmenter 23 whjch indudes of a bank of and , decisjon 
which systematically separates the watermark into several ^ ag shown m FI(J 3 ^ Qu Qf ^ watermark 
subwatermarks. Any portion of the original watermark identifiers are mbols from the al habet used in me ori mal 
might appear redundantly in several of the resulting subwa- 20 encoded watermark si , ^ identifled ^ are reas . 
termarks. Concurrently, the data to be watermarked is used ^ a ^ encoded watennark b me bol 
as an input to data segmenter 24, which segments the data combiner 4J Fma „ ^ [csm eQcoded watermark ^ 
into blocks or subregions, such as 8x8 subimages, as in d d d . h ' corrector 46 
INSERT-MPEG-A. Each of the subwatermarks output by uc ^ aco °y lnc el ™» correcior *o. 
the watermark segmenter 23 is then inserted into a data „ . The insertion and extraction P^^^' 1 ' now be 
block by one of the watermark inserters 25a, 256, etc. The 25 ^ INSERT-ORIGINAL and 
procedure used by the watermark inserters 25a, 256, etc., is EXTRACT-ORIGINAL the object ,s to embed a single PN 
the same procedure described connection with watermark (P 56 ^ 0 random number) sequence mto an image when the 
inserter 18 in FIG. 1. That is, each subwatermark is added on &™ ] lma 8 e mailable at the time of extract.on. The 
into a spectrally transformed data block after spectral 30 Ration associated with the PN sequence is assumed to- 
shaping, and the resulting data is then transformed back into b u e stored m • database together with the original image and 
the spatial domain. Finally, the watermarked data blocks the «P c f» 1 locatlon «* th f embedded watermark. The loca- 
fromthewatermarki n serters25 fl ,25f,,etc.,areassembledby ' 10ns of <he watermarked components has to be recorded 
data combiner 26 to produce watermarked data. b « 5 « nse the implementat.on approbated the N perceptu- 
___ - . ' r tL ally most significant regions of the watermark by the N 
FIG. 3 shows a schematic block diagram of the 35 , , «- . . TT t r . , . * • • . 
rvmA^rumriTr a * \ a ' largest coefficients. However, this ranking was not invariant 
EXTRACT- WHOLE procedure. The watermarked image, . * 1 • « XI1 + «= * * 

j. j . . j • , * 1 to the watermarkmg process. The N largest coefficients may 

video or multimedia data is first used as input into a spectral . ..~ , - . - . , . , f f 

t . . v . , *~ . be different after inserting the watermark than before lnsert- 

normalizer 30 to undo any previously performed spectral ^ ^ e W atennark 

shaping. If the data contains a watermark, then the output of m ^ e ennar * . . 

the spectral normalizer 30 will resemble the spectral trans- 40 , In order t0 av01 ? this problem, the present invention 
formation of the PN coding of that watermark (the signal P laces a watermark in predetermined locations of the 
that was input to the spectral shaper 14 in FIG. 1). The spectrum, typically the first N coefficients. However any 
output of the spectral normalizer 30 is then used as an input predetermined locations could be used, though such loca- 
te several correlators 31a, 31fc etc., which test the water- h ° ns should b *}™Z to the Perceptually significant regions of 
mark with the PN codes used to represent the various 45 the spectrum if the watennark is to survive common signals 
symbols that the encoded watermark might contain (i.e. each transformations such as compression, scaling, etc. 
correlator tests for one PN code that is used to encode a More generally, the information to be embedded is a 
symbol by the PN mapper 11 of FIG. 1). The outputs of the sequence of m symbols drawn from an alphabet A (e.g. the 
correlators 31a, 316, etc., are used as inputs to a decision binary digits or the ASCII symbols). This data is then 
circuit 32, which determines the most likely sequence of 50 supplemented with additional symbols for error detection 
symbols. Finally, this sequence is corrected by an error and correction. Each symbol is then spread spectrum 
corrector 33, which performs the inverse coding that was modulated, a process that maps each symbol into a unique 
performed by the error correction encoder 10 in FIG. 1. The PN sequence known as a chip. The number of bits per chip 
result is the extracted watermark. is preset - the longer the chip length, the higher the detected 
In EXTOACT-MPEG-A, the data from which a water- 55 signal-to-noise ratio will be, but this is at the expense of 
mark is to be extracted is first segmented into several blocks, signaling bandwidth. 

such as 8x8 subimages, exactly as in INSERT-MPEG-A. ™ e power spectrum of the PN sequence is white, i.e. flat, 

The signal from each subimage is then normalized and used and is therefore shaped to match that of the "noise", i.e. the 

as input into a bank of correlators similar to the correlators image/video/audio/or multimedia data into which the water- 

31a, 316, etc. in FIG. 3. The output from the correlators is 60 mark is to embedded. It is this spectral shaping that must 

then averaged with the outputs of corresponding correlators be modified from the prior methods so that the extraction 

from other subimages, and the resulting average correlations process no longer requires the original image. To do this, 

are used as inputs into the decision circuit 32 for subsequent each coefficient of the watermarked spectrum is scaled by 

processing as described above. the local average of the power in the image spectral coef- 

FIG. 4 shows a schematic block diagram of a watermark 65 ficicnt rather lhan ^ coefficient itself, i.e. 
extraction procedure in accordance with the teachings of the 

present invention. The watermarked data is first segmented //-/i+^vgdZ/IW 0) 
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The averaging is the averaging of the absolute coefficient found to improve the detection response and significantly 

values and not the coefficient values themselves. This is reduced the computation requirement associated with each 

effectively estimating the average power present at each block. 

frequency. Other averaging procedures are possible, for In practicing the present invention preferably there is a 

example, averaging over several frames or average of local 5 unique PN sequence for each symbol in the alphabet. The 

neighborhoods of 8x8 blocks. method is relatively robust to clipping since the detector 

This average may be obtained in several ways. It may be output reduces linearly with the quantity of 8x8 subimage 

a local average over a two dimensional region. Alternatively, blocks in the image. For DVD (digital video disk) embedded 

the two dimensional spectrum may be sampled to form a one signaling for APS (analog protection system) and CGMS 

dimensional vector and a one dimensional local average may 10 (copy generation management system), there would be a 

be performed. One dimensional vectorization of the two total of 8 or 16 PN sequences. 

dimensional 8x8 DCT coefficients is already performed as The number of 8x8 blocks in a 512x512 image is 4096, 

part of MPEG II. The average may be a simple box or suggesting that significantly more than one of 16 symbols 

weighted average over the neighborhood. can be embedded in an image or video frame. Assume, for 

For video data, temporal averaging of the spectral coef- is example, that it is desired to embed 1 out of 128 symbols in 

ficients over several frames can also be applied. However, an image. It is necessary to perform 128 parallel correla- 

since several frames are needed for averaging at the spectral tions. This is computationally tractable but hardware imple- 

normalization stage of the extractor, the protection of indi- mentations of each correlation become more complex. An 

vidual video frames taken in isolation may not be possible. alternative method is to only use two binary symbols. It may 

For this reason, the present invention treats video as a very 20 be preferable to associate more than one PN sequence with 

large collection of still images. In this way, even individual each of the two binary symbols or bits in order to increase 

video frames are copy protected. the difficulty of intentionally removing the watermark. In 

In order to extract the watermark, it is necessary to this case, there are only two correlators and a binary string 

perform the spectral normalization, in which the previously may be embedded into the image. The raw bit error rate will 

performed spectral shaping procedure is inverted. In the 25 be very high due by the low detector output. However, this 

present invention, the original unwatermarked signal is not can be reduced to acceptable levels by using error correcting 

available. Thus, the average power of the frequency codes, such as Reed-Solomon (RS). RS codes are robust to 

coefficients, avg(|f j), is approximated by the average of the burst error which may occur because of clipping of the 

watermarked signal, i.e. avg(|f ( '|) image. Other error correcting codes may also be used. 

30 When using this method, it is necessary for the receiver to 

»vguV„ f <//D (2) know the start location of the encoded block. The start 

t, . . t , , . /xewwj u location may not be obvious, particularly when the image 

This is approximately true since oiavg( f j)W- ff , where . . f. . , , . * „ J 6 

\w • a. ♦ i -j * . * * • ii nas b eeD subjected to clipping. However, convention syn- 

W, is the watermark component, and ais a constant typically « ■ . *V, , {* , . J , 

. u ^ n t j a Ai chronizing methods can be used; such as preceding each 

in the range between 0.1 and 0.01. „ , ■ i • . i *■ r u i 

™ 6 . ., . ™ . 35 block with a special or unique symbol or string of symbols. 

The normalization stage then divides each coefficient (f. f ) ~ . » i i_ o o li i ■ * . j 

. . j . i u »u i i i\c ii\ ■ \t To insert a watermark, each 8x8 block is treated as an 

nei hborhood * individual subimage or subregion. The DCT of the subimage 

k * is then computed and the two dimensional DCT is vectorized 

' in the zigzag pattern shown in FIG. 5, although other 

40 patterns are also possible. These two stages constitute most 

57171) = aJff\) ' of the calculations but are P art of the MPEG encoding 

av * ' ° vg process. Next, a PN noise sequence {v/ 1 . . . w n } is inserted 

^ ft +ffWi into the DCT coefficients using Equation 1 as before. The 

av s(\fi\) ' length of the PN sequence cannot exceed 64 (in an 8x8 

45 block) and is typically much shorter, in the range of 11 to 25. 

Hie first term, on the right hand side (RHS) of Equation l [ onl l a single code is to be inserted into the image, 

then the same PN sequence is inserted into each of the 
720 x 480/64 -5400 blocks. However, a variation may be 
j. performed at this point in the procedure. Within each row of 

avg Qfi\y 50 blocks, the PN sequence is cyclically rotated by one fre- 

quency coefficient prior to insertion in the subsequent block. 
Similarly, the PN sequence is cyclically rotated by one 
is considered a noise term. This term was not present in the frequency coefficient at the start of each new row. FIG. 6 
system described in U.S. patent application Sen No. 08/534, illustrates an order of rotations. 

894, because access to the unwatermarked coefficients 55 The purpose of these rotations or shifts is to improve the 
allowed this term to be removed. The second term aW. is the response of the watermark extraction stage. Earlier experi- 
original watermark signal which can now be detected using ments revealed that certain DCT coefficients were more 
conventional correlation. difficult to estimate than others. The location at these coef- 

If the watermark is extracted from any single 8x8 block, ficients varied from image to image. However, within an 
the detector reliability is very low. If, however, the water- 60 image, the coefficient could be consistently poor, 
marks extracted from each 8x8 block are first added together Consequently, without shifting, one or more of the estimated 
and the averaged watermark is then applied to the correlator, watermark coefficients could be significantly degraded rela- 
then a very strong and unambiguous response is obtained. tive to the other watermark coefficients, thereby reducing the 
This differs from the method described in U.S. patent detector performance. Conversely, shifting significantly 
application Ser. No. 08/715,953 in which correlation 65 reduces the effect a poor DCT coefficients has on a single 
occurred within each block and the output from each corr- watermark coefficient and the detector performance is mark- 
clator was averaged together. The present invention was edly improved. Note that any cyclic pattern can be used. 
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Further modifications are useful once rotation of the 
watermark has been introduced. First, the length of the 
watermark may now be significantly greater than 64 .Then, 
for each block only a small subset of the watermark (say 
five) coefficients is inserted into the first five DCT coeffi- 
cients (excluding the d.c. term). Because of the rotation, a 
different subset of the watermark is inserted into neighbor- 
ing 8x8 blocks. Finally, having completed the watermark 
insertion, the MPEG encoder is able to proceed with the 
subsequent stages of compression. 

Note that the watermark may also be inserted after the 
MPEG quantization stage to reduce distortion of the water- 
mark. MPEG-2 performs a convenient one dimension vec- 
torization called "zigzagging", which allows a simple 3x1 
box average to be performed on the coefficients (excluding 
the d.c. term). 

In practice, performance was improved if the averaging is 
performed using the 2 four-connected coefficients closest to 
the d.c. term, as illustrated in FIG. 7, i.e. the two coefficients 
above and to the left. 

Watermark detection begins by first extracting the PN 
noise sequence from each 8x8 block using Equation l.For 
each block, the PN sequence is then cyclically shifted in the 
opposite direction by one frequency coefficient, and the 
average over all the blocks is then computed. In practice, this 
process can be computed incrementally and does not require 
temporary storage of all the extracted watermarks. A 
weighted averaging can also be applied, where the weights 
are determined based on their susceptibility to common 
signal transformations such as low pass filtering. Finally, the 
average watermark is compared with the original PN 
sequence via correlation. The reason for shifting the water- 
mark in the column direction may now be apparent. If the 
image is clipped on an arbitrary block boundary, then the 
computed average watermark will simply be rotated by an 
amount that is a function of the relative location of the 
clipped portion of the image. Correlation can then be per- 
formed on all permutations (typically 11 to 25) of the 
watermark. The output from the correlator with the maxi- 
mum value is then used for decision purposes. The extrac- 
tion stage is depicted in FIG. 4. Taking the maximum 
correlator output over all rotations of the watermark can 
cause the decision circuitry to be noisy. To improve this, the 
watermark is broken into two pieces; a synchronization 
portion is of length K and a verification portion is N-K. 
Then, when the watermark is extracted as before, correlation 
is first performed only on all rotations of the synchronization 
portion of this watermark. The maximum correlation output 
is noted, then the verification portion of the watermark is 
rotated by the corresponding amount and a second correla- 
tion is performed on the verification portions of the water- 
marks. This process significantly improves the overall reli- 
ability of the system. In the course of experimentation, it was 
noticed that some watermarks performed better than others 
on the same imagery. This was caused by variation in the 
correlation statistics between the synchronization and veri- 
fication portions of the watermark. Ideally, the two portions 
should have very low correlations. However, in several cases 
where watermarks performed poorly, it was traced to unex- 
pected correlations between the two portions. 

The present invention provides a modification to digital 
watermarking methods in which the original data is required 
for watermark extraction thereby enabling watermarking 
extraction in the absence of an unwatermarked or original 
data. The present invention preferably uses MPEG/JPEG 
coefficients. An image is divided into typically 8x8 block 
subimages or subregions and each subimage is processed 



and the results are combined to derive the extracted water- 
mark. The result is extraction of the watermark with very 
high confidence. 

While the above invention describes improvements to the 
prior-art INSERT- WHOLE, INSERT-MPEG-A, 
EXTRACT-WHOLE, and EXTRACT-MPEG-A algorithms, 
it should be apparent to anyone skilled in the art that the 
same improvements may be applied to any algorithm for 
inserting and extracting watermarks in image data. This 
more general view of the present invention is shown in 
FIGS. 8 and 9. 

FIG. 8 shows a schematic block diagram of the general 
method for inserting watermarks. This general method 
makes use of a non-block-based watermark insertion 
algorithm, which shall be referred to hereafter as the "base 
insertion algorithm". The watermark encoder 80 converts 
the watermark into a form appropriate for the base insertion 
algorithm. If the base insertion algorithm is that shown in 
FIG. 1, for example, then the watermark encoder 80 corre- 
sponds to the watermark encoder 19, which comprises the 
error correction encoder 10, the PN mapper 11, and the 
spectral transformer 12. However, if a different base inser- 
tion algorithm is to be used, then the watermark encoder 80 
may perform a different transformation of the watermark. 
The encoded watermark signal from watermark encoder 80 
is provided as an input to watermark segmenter 81, which 
divides the watermark into a set of subwatermarks. Any 
portion of the original watermark might appear redundantly 
in several of the resulting subwatermarks. The data to be 
watermarked is provided as an input to data segmenter 82, 
which divides the data into subregions. Each subwatermark 
is inserted into a respective data subregion by a watermark 
inserter 83a, 836, etc. The watermark inserters implement 
the base insertion algorithm, so, if the base insertion algo- 
rithm is that shown in FIG. 1, then each watermark inserter 
83a, 836, etc, corresponds to the watermark inserter 18, 
which comprises a spectral transformer 13, a spectral shaper 
14, a delay 15, a summer 16, and an inverse transform 17. 
However, if a different base insertion algorithm is to be used, 
40 then the watermark inserters 83a, 836, etc., may employ a 
different method of inserting subwatermarks into the subre- 
gions of the data to be watermarked. The outputs from the 
watermark inserters are assembled in data combiner 84 to 
provide watermarked data. 

FIG. 9 shows a schematic block diagram of the corre- 
sponding general extraction algorithm. The algorithm makes 
use of a "base extraction" algorithm that corresponds to the 
base insertion algorithm used in inserting the watermark into 
the data to be watermarked (FIG. 8). The algorithm in FIG. 
9 is substantially the same as the algorithm shown in FIG. 
4, except that, in the general case, the spectrum normalizers 
41a, etc. are replaced by watermark extractors 91a, etc., 
which implement the base extraction algorithm. That is, if 
the base insertion algorithm used was the algorithm shown 
in FIG. 1, then the watermark extractors 91a, etc., in FIG. 9 
will be the spectrum normalizers 41a, etc. in FIG. 4. 

While there has been described and illustrated a system 
for inserting a watermark into and extracting a watermark 
from watermarked data without using an unwatermarked 
version of the data, it will be apparent to those skilled in the 
art that variations and modifications are possible without 
deviating from the broad principles and teachings of the 
present invention which shall be limited solely by the scope 
of the claims appended hereto. 
What is claimed is: 

1. A method for inserting a watermark signal into data to 
be watermarked comprising the steps of: 
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dividing data to be watermarked into a plurality of sub- 
regions; 

computing frequency coefficients of the data to be water- 
marked in each subregion; 

spread spectrum modulating a watermark signal to be 
inserted by mapping the watermark signal into a PN 
(pseudo-random noise) sequence; 

spectral shaping the PN sequence as a function of the 
average power in each frequency coefficient of the data; 
and 

inserting each spectral shaped PN sequence into prede- 
termined coefficients in the data in each subregion. 

2. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 1, where said inserting 
is performed after the data undergoes MPEG quantization 
processing. 

3. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 1, where said frequency 
coefficients are DCT (discrete cosine transform) coefficients. 

4. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 3, where each subre- 
gion is a 8x8 block of pixels. 

5. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 4, where said inserting 
is performed after the data undergoes MPEG quantization 
processing. 

6. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 1, where each subre- 
gion is a 8x8 block of pixels. 

7. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 6, where said inserting 
is performed after the data undergoes MPEG quantization 
processing. 

8. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 6, where the frequency 
coefficients of the watermark signal are rotated prior to 
inserting of each spectral shaped PN sequence into the 
subregion. 

9. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 8, where said inserting 
is performed after the data undergoes MPEG quantization 
processing. 

10. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 8, where only a subset 
of the watermark signal frequency coefficients is inserted 
into any one subregion. 

11. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 10, where the water- 
mark signal comprises a synchronization portion and a 
verification portion. 

12. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 11, where said inserting 
is performed after the data undergoes MPEG quantization 
processing. 

13. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 11, where the synchro- 
nization portion and the verification portion have very little 
correlation between each other. 

14. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 1, where the spectral 
shaping as a function of the average power is typically 3x1 
window of the coefficient obtained from the one- 
dimensional vectorization by zigzagging of two-dimension 
frequency coefficients. 

15. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 1, where the spectral 
shaping is a function of the average power based on the two 
four-connected frequency coefficients closest to the DC 
term. 



5,027 

12 

16. A method of extracting a watermark from water- 
marked data comprising the steps of: 

receiving subregions of watermarked data; 
spectrum normalizing the watermarked data as a function 
5 of the average power in each frequency coefficient of 

the watermarked data in each subregion to generate 

respective normalized signals; 
combining the respective normalized signals from each 

subregion to generate a single watermark; 
10 correlating the single watermark with predetermined PN 

(pseudo-random noise) sequences corresponding to 

predetermined symbols to provide correlated signals 

for each predetermined PN sequence in each subregion; 
deciding which correlated signal is most likely a current 
25 symbol; and 

extracting a sequence of most likely current symbols 

corresponding to the watermark. 

17. A method of extracting a watermark from water- 
marked data as set forth in claim 16, where the subregions 

20 are 8x8 blocks used for MPEG encoding and decoding. 

18. A method of extracting a watermark from water- 
marked data as set forth in claim 17, where said combining 
the normalized signals from each subregion to generate a 
single watermark, including removing the relative rotation 
of the watermark between blocks. 

25 19. A method of extracting a watermark from water- 
marked data as set forth in claim 18, further comprising 
subsequently reconstructing the watermark from partial 
watermarks inserted into each block. 
20. A method of extracting a watermark from water- 

30 marked data as set forth in claim 19, further comprising 
weighting the watermark coefficients based on their location 
within the frequency spectrum, where the weighting is a 
function of the susceptibility of each frequency coefficient to 
common signal transformations. 

35 21. A method of extracting a watermark from water- 
marked data as set forth in claim 16, further comprising 
correlating with all rotational shifts of the extracted water- 
mark and selecting the maximum value. 

22. A method of extracting a watermark from water- 
40 marked data as set forth in claim 16, further comprising 

correlating with all rotational shifts of a synchronization 
portion of a watermark to determine a maximum value and 
subsequently rotating a verification portion of the watermark 
by the same amount as the synchronization portion is rotated 
45 to obtain the maximum value prior to correlating between 
the verification portion and predetermined PN sequences. 

23. A method of extracting a watermark from water- 
marked data comprising the steps of: 

5Q receiving subregions of watermarked data; 

spectrum normalizing the watermarked data as a function 
of the average power in each frequency coefficient of 
the watermarked data in each subregion to generate 
respective normalized signals; 
55 correlating the respective normalized signals with prede- 
termined PN sequences corresponding to predeter- 
mined symbols to provide correlated signals for each 
predetermined PN sequence in each subregion; 
deciding which correlated signal is most likely a current 
60 symbol in each subregion for providing an extracted 
symbol stream; 
error correcting the extracted symbol stream; and 
extracting a sequence of most likely current symbols 
corresponding to the watermark. 
65 24. A method of extracting a watermark from water- 
marked data as set forth in claim 23, where said error 
correction is Reed Solomon error correction. 
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25. A method for inserting a watermark signal into data to 
be watermarked comprising the steps of: 

dividing data to be watermarked into a plurality of sub- 
regions; 

dividing a watermark signal into a plurality of subwater- 
marks where portions of the watermark are contained in 
more than one subwatermark; and 

inserting said plurality of subwatermarks into said plu- 
rality of subregions. 

26. A method for inserting a watermark signal into data to 
be watermarked as set forth in claim 25, where each sub- 
watermark is inserted into a respective subregion, so that 
each subregion contains at least one subwatermark. 

27. A method for extracting a watermark signal from 
watermarked data comprising the steps of: 

receiving a plurality of subregions of watermark data; 
extracting a subwatermark from each subregion of said 
plurality of subregions; and 
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combining and averaging the subwatermarks extracted 
from all the subregions to obtain a signal commensu- 
rate with the watermark signal. 

28. A method for extracting a watermark signal from 
watermarked data as set forth in claim 27, further compris- 
ing the steps of: 

dividing the signal commensurate with the watermark 
signal into a plurality of symbol signals; 

correlating each symbol signal with a set of predefined 
signals; 

determining which predefined signal best corresponds to 
each symbol signal; and 

combining the best corresponding predetermined signals 
to generate the watermark signal. 
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